Manticore Search is a multi-storage database specifically designed for search, with robust full-text search capabilities.

As an open-source database (available on GitHub), Manticore Search was created in 2017 as a continuation of Sphinx Search engine. Our development team took all the best features of Sphinx and significantly improved its functionality, fixing hundreds of bugs along the way (as detailed in our Changelog). With nearly complete code rewrites, Manticore Search is now a modern, fast, and light-weight database with full features and exceptional full-text search capabilities.

Manticore's key features are:

Powerful and fast full-text searching that works well for small and large datasets

• Over 20 full-text operators and over 20 ranking factors
• Custom ranking
• Stemming
• Lemmatization
• Stopwords
• Synonyms
• Wordforms
• Advanced tokenization at character and word level
• Proper Chinese segmentation
• Text highlighting

Vector search capabilities

Manticore Search supports the ability to add embeddings generated by your Machine Learning models to each document, and then doing a nearest-neighbor search on them. This lets you build features like similarity search, recommendations, semantic search, and relevance ranking based on NLP algorithms, among others, including image, video, and sound searches.

Multithreading

Manticore Search utilizes a smart query parallelization to lower response time and fully utilize all CPU cores when needed.

Cost-based query optimizer

The cost-based query optimizer uses statistical data about the indexed data to evaluate the relative costs of different execution plans for a given query. This allows the optimizer to determine the most efficient plan for retrieving the desired results, taking into account factors such as the size of the indexed data, the complexity of the query, and the available resources.

Storage options

Manticore offers both row-wise and column-oriented storage options to accommodate datasets of various sizes. The traditional and default row-wise storage option is available for datasets of all sizes - small, medium, and large, while the columnar storage option is provided through the Manticore Columnar Library for even larger datasets. The key difference between these storage options is that row-wise storage requires all attributes (excluding full-text fields) to be kept in RAM for optimal performance, while columnar storage does not, thus offering lower RAM consumption, but with a potential for slightly slower performance (as demonstrated by the statistics on https://db-benchmarks.com/).

Automatic secondary indexes

Manticore Columnar Library uses Piecewise Geometric Model index, which exploits a learned mapping between the indexed keys and their location in memory. The succinctness of this mapping, coupled with a peculiar recursive construction algorithm, makes the PGM-index a data structure that dominates traditional indexes by orders of magnitude in space while still offering the best query and update time performance. Secondary indexes are ON by default for all numeric fields.

SQL-first

Manticore's native syntax is SQL and it supports SQL over HTTP and MySQL protocol, allowing for connection through popular mysql clients in any programming language.

JSON over HTTP

For a more programmatic approach to managing data and schemas, Manticore provides HTTP JSON protocol, similar to that of Elasticsearch.

Elasticsearch-compatible writes

You can execute Elasticsearch-compatible insert and replace JSON queries which enables using Manticore with tools like Logstash (version < 7.13), Filebeat and other tools from the Beats family.

Declarative and imperative schema management

Easily create, update, and delete tables online or through a configuration file.

The benefits of C++ and the convenience of PHP

The Manticore Search daemon is developed in C++, offering fast start times and efficient memory utilization. The utilization of low-level optimizations further boosts performance. Another crucial component, called Manticore Buddy, is written in PHP and is utilized for high-level functionality that does not require lightning-fast response times or extremely high processing power. Although contributing to the C++ code may pose a challenge, adding a new SQL/JSON command using Manticore Buddy should be a straightforward process.

Real-Time inserts

Newly added or updated documents can be immediately read.

Interactive courses for easy learning

We offer free interactive courses to make learning effortless.

Transactions

While Manticore is not fully ACID-compliant, it supports isolated transactions for atomic changes and binary logging for safe writes.

Built-In replication and load balancing

Data can be distributed across servers and data centers with any Manticore Search node acting as both a load balancer and a data node. Manticore implements virtually synchronous multi-master replication using the Galera library, ensuring data consistency across all nodes, preventing data loss, and providing exceptional replication performance.

Built-in backup capabilities

Manticore is equipped with an external tool manticore-backup, and the BACKUP SQL command to simplify the process of backing up and restoring your data. Alternatively, you can use mysqldump to make logical backups.

Out-of-the-box data sync

The indexer tool and comprehensive configuration syntax of Manticore make it easy to sync data from sources like MySQL, PostgreSQL, ODBC-compatible databases, XML, and CSV.

Integration options

You can integrate Manticore Search with a MySQL/MariaDB server using the FEDERATED engine or via ProxySQL.

You can use Apache Superset and Grafana to visualize data stored in Manticore. Various MySQL tools can be used to develop Manticore queries interactively, such as HeidiSQL and DBForge.

Stream filtering made easy

Manticore offers a special table type, the "percolate" table, which allows you to search queries instead of data, making it an efficient tool for filtering full-text data streams. Simply store your queries in the table, process your data stream by sending each batch of documents to Manticore Search, and receive only the results that match your stored queries.

Possible applications:

Manticore has a variety of use cases, including:

• Full-text search
  • With small data volumes, enjoy the benefits of powerful full-text search syntax and low memory consumption (as low as 7-8 MB).
  • With large data, benefit from Manticore's high availability and ability to handle massive tables.
• OLAP: Use Manticore Search and the Manticore Columnar Library to analyze terabytes of data on a single or multiple servers.
• Faceted search
• Geo-spatial search
• Spell correction
• Autocomplete
• Data stream filtering

Requirements

• Architecture: arm64 or x86_64
• OS: Debian-based (e.g. Debian, Ubuntu, Mint), RHEL-based (e.g. RHEL, CentOS, Alma, Oracle Linux, Amazon Linux), Windows, or MacOS.
• Manticore Columnar Library, which provides columnar storage and secondary indexes, requires a CPU with SSE >= 4.2.
• No specific disk space or RAM requirements are needed. An empty Manticore Search instance only uses around 40MB of RSS RAM.

About this manual

The manual is arranged to reflect the most likely way you would use Manticore:

• starting with basic information about it and how to install and connect
• essential things like adding documents and running searches
• performance optimization tips, tricks and extending Manticore with the help of plugins and custom functions

Do not skip 1️⃣ 2️⃣ 3️⃣

Key sections of the manual are marked with 1️⃣, 2️⃣, 3️⃣ etc. in the menu for your convenience since their corresponding functionality is most used. If you are new to Manticore we highly recommend not skipping them.

Quick start guide

If you are looking for a quick understanding of how Manticore works in general ⚡ Quick start guide is a good place to start.

Using examples

Each query example has a little icon 📋 in the top-right corner:

You can use it to copy examples to the clipboard. If the query is an HTTP request it will be copied as a CURL command. You can configure the host/port if you press ⚙️.

Search in this manual

We love search and we've made our best to make searching in this manual as convenient as possible. Of course it's backed by Manticore Search. Besides using the search bar which requires opening the manual first there is a very easy way to find something by just opening mnt.cr/your-search-keyword :

Best practices

There are few things you need to understand about Manticore Search that can help you follow the best practices of using it.

Real-time table vs plain table

• Real-time table allows adding, updating and deleting documents with immediate availability of the changes.
• Plain table is a mostly immutable data structure and a basic element used by real-time tables. Plain table stores a set of documents, their common dictionary and indexation settings. One real-time table can consist of multiple plain tables (chunks), but besides that Manticore provides direct access to building plain tables using tool indexer. It makes sense when your data is mostly immutable, therefore you don't need a real-time table for that.

Real-time mode vs plain mode

Manticore Search works in two modes:

• Real-time mode (RT mode). This is a default one and allows to manage your data schema imperatively:
• allows managing your data schema online using SQL commands CREATE/ALTER/DROP TABLE and their equivalents in non-SQL clients
• in the configuration file you need to define only server-related settings including data_dir
• Plain mode allows to define your data schemas in a configuration file, i.e. provides declarative kind of schema management. It makes sense in three cases:
• when you only deal with plain tables
• or when your data schema is very stable and you don't need replication (as it's available only in the RT mode)
• when you have to make your data schema portable (e.g. for easier deployment of it on a new server)

You cannot combine the 2 modes and need to decide which one you want to follow by specifying data_dir in your configuration file (which is the default behaviour). If you are unsure our recommendation is to follow the RT mode as if even you need a plain table you can build it with a separate plain table config and import to your main Manticore instance.

Real-time tables can be used in both RT and plain modes. In the RT mode a real-time table is defined with a CREATE TABLE command, while in the plain mode it is defined in the configuration file. Plain (offline) tables are supported only in the plain mode. Plain tables cannot be created in the RT mode, but existing plain tables made in the plain mode can be converted to real-time tables and imported in the RT mode.

SQL vs JSON

Manticore provides multiple ways and interfaces to manage your schemas and data, but the two main are:

• SQL. This is a native Manticore's language which enables all Manticore's functionality. The best practice is to use SQL to:
• manage your schemas and do other DBA routines as it's the easiest way to do that
• design your queries as SQL is much closer to natural language than the JSON DSL which is important when you design something new. You can use Manticore SQL via any MySQL client or /sql.
• JSON. Most functionality is also available via JSON domain specific language. This is especially useful when you integrate Manticore with your application as with JSON you can do it more programmatically than with SQL. The best practice is to first explore how to do something via SQL and then use JSON to integrate it into your application.

Docker images of Manticore Search are publicly accessible on Docker Hub, built from the Manticore Search docker GitHub repository.

To retrieve the Manticore image, run the following command:

docker pull manticoresearch/manticore

For more information about using Manticore in Docker, see the Using Manticore in Docker section.

Supported releases:

• CentOS 7, RHEL 7, Oracle Linux 7
• CentOS 8, RHEL 8, Oracle Linux 8, CentOS Stream 8
• Amazon Linux 2
• CentOS 9, RHEL 9, AlmaLinux 9

YUM repository

The simplest method to install Manticore on RedHat/CentOS is by using our YUM repository:

Install the repository:

sudo yum install https://repo.manticoresearch.com/manticore-repo.noarch.rpm

Then install Manticore Search:

sudo yum install manticore manticore-extra

If you are upgrading to Manticore 6 from an older version, it is recommended to remove your old packages first to avoid conflicts caused by the updated package structure:

sudo yum remove manticore*

It won't remove your data and configuration file.

Development packages

If you prefer "Nightly" (development) versions do:

sudo yum -y install https://repo.manticoresearch.com/manticore-repo.noarch.rpm && \
sudo yum -y --enablerepo manticore-dev install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-debuginfo manticore-tools-debuginfo manticore-columnar-lib-debuginfo  manticore-icudata manticore-galera manticore-galera-debuginfo

Standalone RPM packages

To download standalone RPM files from the Manticore repository, follow the instructions available at https://manticoresearch.com/install/.

More packages you may need

For indexer

If you plan to use indexer to create tables from external sources, you'll need to make sure you have installed corresponding client libraries in order to make available of indexing sources you want. The line below will install all of them at once; feel free to use it as is, or to reduce it to install only libraries you need (for only mysql sources - just mysql-libs should be enough, and unixODBC is not necessary).

sudo yum install mysql-libs postgresql-libs expat unixODBC

In CentOS Stream 8 you may need to run:

dnf install mariadb-connector-c

if you get error sql_connect: MySQL source wasn't initialized. Wrong name in dlopen? trying to build a plain table from MySQL.

Ukrainian lemmatizer

The lemmatizer requires Python 3.9+. Make sure you have it installed and that it's configured with --enable-shared.

Here's how to install Python 3.9 and the Ukrainian lemmatizer in Centos 7/8:

# install Manticore Search and UK lemmatizer from YUM repository
yum -y install https://repo.manticoresearch.com/manticore-repo.noarch.rpm
yum -y install manticore manticore-lemmatizer-uk

# install packages needed for building Python
yum groupinstall "Development Tools" -y
yum install openssl-devel libffi-devel bzip2-devel wget -y

# download, build and install Python 3.9
cd ~
wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz
tar xvf Python-3.9.2.tgz
cd Python-3.9*/
./configure --enable-optimizations --enable-shared
make -j8 altinstall

# update linker cache
ldconfig

# install pymorphy2 and UK dictionary
pip3.9 install pymorphy2[fast]
pip3.9 install pymorphy2-dicts-uk

Supported releases:

• Debian
• 10.0 (Buster)
• 11.0 (Bullseye)

• 12.0 (Bookworm)

• Ubuntu

• 18.04 (Bionic)
• 20.04 (Focal)
• 21.04 (Hirsute Hippo)

• 22.04 (Ubuntu Jammy)

• Mint

• 19
• 20

APT repository

The easiest way to install Manticore in Ubuntu/Debian/Mint is by using our APT repository.

Install the repository:

wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update

(install wget if it's not installed; install gnupg2 if apt-key fails).

Then install Manticore Search:

sudo apt install manticore manticore-extra

If you are upgrading to Manticore 6 from an older version, it is recommended to remove your old packages first to avoid conflicts caused by the updated package structure:

sudo apt remove manticore*

It won't remove your data and configuration file.

Development packages

If you prefer "Nightly" (development) versions do:

wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && \
sudo dpkg -i manticore-dev-repo.noarch.deb && \
sudo apt -y update && \
sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l manticore-galera manticore-galera-dbgsym

Standalone DEB packages

To download standalone DEB files from the Manticore repository, follow the instructions available at https://manticoresearch.com/install/.

More packages you may need

For indexer

Manticore package depends on zlib and ssl libraries, nothing else is strictly required. However, if you plan to use indexer to create tables from external storages, you'll need to install appropriate client libraries. To find out what specific libraries indexer requires, run it and look at the top of its output:

$ sudo -u manticore indexer
Manticore 3.5.4 13f8d08d@201211 release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

Built by gcc/clang v 5.4.0,

Built on Linux runner-0277ea0f-project-3858465-concurrent-0 4.19.78-coreos #1 SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux

Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=xenial -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_ICU=1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_POSTGRESQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SOVERSION=31 -DSYSCONFDIR=/etc/manticoresearch

Here you can see mentions of libodbc.so.2, libexpat.so.1, libmysqlclient.so.20, and libpq.so.5.

Below is a reference table with a list of all the client libraries for different Debian/Ubuntu versions:

To find packages that provide the libraries, you can use, for example, apt-file:

apt-file find libmysqlclient.so.20
libmysqlclient20: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
libmysqlclient20: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.2.0
libmysqlclient20: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.6

Note that you only need libraries for the types of storages you're going to use. So if you plan to build tables only from MySQL, then you might need to install only the MySQL library (in the above case libmysqlclient20).

Finally, install the needed packages:

sudo apt-get install libmysqlclient20 libodbc1 libpq5 libexpat1

If you aren't going to use the indexer tool at all, you don't need to find and install any libraries.

To enable CJK tokenization support, the official packages contain binaries with embedded ICU library and include ICU data file. They are independent from any ICU runtime library which might be available on your system, and can't be upgraded.

Ukrainian lemmatizer

The lemmatizer requires Python 3.9+. Make sure you have it installed and that it's configured with --enable-shared.

Here's how to install Python 3.9 and the Ukrainian lemmatizer on Debian and Ubuntu:

# install Manticore Search and UK lemmatizer from APT repository
cd ~
wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt -y update
sudo apt -y install manticore manticore-lemmatizer-uk

# install packages needed for building Python
sudo apt -y update
sudo apt -y install wget build-essential libreadline-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev

# download, build and install Python 3.9
cd ~
wget https://www.python.org/ftp/python/3.9.4/Python-3.9.4.tgz
tar xzf Python-3.9.4.tgz
cd Python-3.9.4
./configure --enable-optimizations --enable-shared
sudo make -j8 altinstall

# update linker cache
sudo ldconfig

# install pymorphy2 and UK dictionary
sudo pip3.9 install pymorphy2[fast]
sudo pip3.9 install pymorphy2-dicts-uk

Via Homebrew package manager

brew install manticoresoftware/tap/manticoresearch manticoresoftware/tap/manticore-extra

Start Manticore as a brew service:

brew services start manticoresearch

The default configuration file for Manticore is located at either /usr/local/etc/manticoresearch/manticore.conf or /opt/homebrew/etc/manticoresearch/manticore.conf.

If you plan to use indexer to fetch data from sources such as MySQL, PostgreSQL, or another database using ODBC, you may need additional libraries, such as mysql@5.7, libpq, and unixodbc, respectively.

Development packages

If you prefer "Nightly" (development) versions do:

brew tap manticoresoftware/tap-dev
brew install manticoresoftware/tap-dev/manticoresearch-dev manticoresoftware/tap-dev/manticore-extra-dev
brew services start manticoresearch-dev

1. Download the Manticore Search Installer and run it. Follow the installation instructions.
2. Choose the directory to install to.
3. Select the components you want to install. We recommend installing all of them.
4. Manticore comes with a preconfigured manticore.conf file in RT mode. No additional configuration is required.

Installing as a Windows service

To install searchd (Manticore Search server) as a Windows service, run:

\path\to\searchd.exe --install --config \path\to\config --servicename Manticore

Make sure to use the full path of the configuration file, otherwise searchd.exe will not be able to locate it when it starts as a service.

After installation, the service can be started from the Services snap-in of the Microsoft Management Console.

Once started, you can access Manticore using the MySQL command line interface:

mysql -P9306 -h127.0.0.1

Note that in most examples in this manual, we use -h0 to connect to the local host, but in Windows, you must use localhost or 127.0.0.1 explicitly.

Compiling Manticore Search from sources enables custom build configurations, such as disabling certain features or adding new patches for testing. For example, you may want to compile from sources and disable the embedded ICU in order to use a different version installed on your system that can be upgraded independently of Manticore. This is also useful if you are interested in contributing to the Manticore Search project.

Building using CI Docker

To prepare official release and development packages, we use Docker and a special building image. This image includes essential tooling and is designed to be used with external sysroots, so one container can build packages for all operating systems. You can build the image using the Dockerfile and README or use an image from Docker Hub. This is the easiest way to create binaries for any supported operating system and architecture. You'll also need to specify the following environment variables when running the container:

• DISTR: the target platform
• arch: the architecture
• SYSROOT_URL: the URL to the system roots archives. You can use https://repo.manticoresearch.com/repository/sysroots unless you are building the sysroots yourself (instructions can be found here).
• Use the CI workflow files as a reference to find the other environment variables you might need to use:
• https://github.com/manticoresoftware/manticoresearch/blob/master/.github/workflows/pack_publish.yml
• https://github.com/manticoresoftware/manticoresearch/blob/master/.github/workflows/build_template.yml

To find possible values for DISTR and arch, you can use the directory https://repo.manticoresearch.com/repository/sysroots/roots_with_zstd/ as a reference, as it includes sysroots for all supported combinations.

After that, building packages inside the Docker container is as easy as calling:

cmake -DPACK=1 /path/to/sources
cmake --build .

For instance, to create a package for RHEL7-compatible operating systems that is similar to the official version Manticore Core Team provides, you should execute the following commands in the directory containing the Manticore Search sources. This directory is the root of a cloned repository from https://github.com/manticoresoftware/manticoresearch:

docker run -it --rm -e SYSROOT_URL=https://repo.manticoresearch.com/repository/sysroots \
-e arch=x86_64 \
-e DISTR=rhel7 \
-e boost=boost_rhel_feb17 \
-e sysroot=roots_nov22 \
-v $(pwd):/manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
manticoresearch/external_toolchain:clang16_cmake3263 bash

# following is to be run inside docker shell
cd /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/
mkdir build && cd build
cmake -DPACK=1 ..

cmake --build .
# or if you want to build packages:
# cmake --build . --target package

The long source directory path is required or it may fail to build the sources.

The same process can be used to build binaries/packages not only for popular Linux distributions, but also for FreeBSD, Windows, and macOS.

Building manually

Compiling Manticore without using the building Docker is not recommended, but if you need to do it, here's what you may need to know:

Required tools

• C++ compiler
• In Linux - GNU (4.7.2 and above) or Clang can be used
• In Windows - Microsoft Visual Studio 2019 and above (community edition is enough)
• On macOS - Clang (from command line tools of XCode, use xcode-select --install to install).
• Bison, Flex - on most systems, they are available as packages, on Windows they are available in the cygwin framework.
• Cmake - used on all platforms (version 3.19 or above required)

Fetching sources

From git

Manticore source code is hosted on GitHub.
To obtain the source code, clone the repository and then check out the desired branch or tag. The branch master represents the main development branch. Upon release, a versioned tag is created, such as 3.6.0 and a new branch for the current release is started, in this case manticore-3.6.0. The head of the versioned branch after all changes is used as source to build all binary releases. For example, to take sources of version 3.6.0 you can run:

git clone https://github.com/manticoresoftware/manticoresearch.git
cd manticoresearch
git checkout manticore-3.6.0

From archive

You can download the desired code from GitHub by using the "Download ZIP" button. Both .zip and .tar.gz formats are suitable.

wget -c https://github.com/manticoresoftware/manticoresearch/archive/refs/tags/3.6.0.tar.gz
tar -zxf 3.6.0.tar.gz
cd manticoresearch-3.6.0

Configuring

Manticore uses CMake. Assuming you are inside the root directory of the cloned repository:

mkdir build && cd build
cmake ..

CMake will investigate available features and configure the build according to them. By default, all features are considered enabled if they are available. The script also downloads and builds some external libraries, assuming that you want to use them. Implicitly, you get support for the maximal number of features.

You can also configure the build explicitly with flags and options. To enable feature FOO add -DFOO=1 to the CMake call.
To disable it, use -DFOO=0. If not explicitly noted, enabling a feature that is not available((such as WITH_GALERA on an MS Windows build)) will cause the configuration to fail with an error. Disabling a feature, apart from excluding it from the build, also disables its investigation on the system and disables the downloading/building of any related external libraries.

Configuration flags and options

• USE_SYSLOG - allows the use ofsyslog in query logging.
• WITH_GALERA -Enables support for replication on the search daemon. Support will be configured for the build, and the sources for the Galera library will be downloaded, built, and included in the distribution/installation. Usually, it is safe to build with Galera, but not distribute the library itself (so no Galera module, no replication). However, sometimes you may need to explicitly disable it, such as if you want to build a static binary that by design cannot load any libraries, so that even the presence of a call to the 'dlopen' function inside the daemon will cause a link error.
• WITH_RE2 - Builds with the use of the RE2 regular expression library. This is necessary for functions like REGEX(), and the regexp_filter
  feature.
• WITH_RE2_FORCE_STATIC - Downloads the sources of RE2, compiles them, and links them statically, so that the final binaries will not depend on the presence of a shared RE2 library in your system.
• WITH_STEMMER - Builds with the use of the Snowball stemming library.
• WITH_STEMMER_FORCE_STATIC - Downloads the Snowball sources, compiles them, and links them statically, so that the final binaries will not depend on the presence of a shared libstemmer library in your system.
• WITH_ICU - Builds with the use of ICU, the International Components for Unicode library. This is used in tokenization of Chinese, for text segmentation. This is in play when morphology like icu_chinese is in use.
• WITH_ICU_FORCE_STATIC - Downloads the ICU sources, compiles them, and links them statically, so that the final binaries will not depend on the presence of a shared icu library in your system. Also includes the ICU data file into the installation/distribution. The purpose of a statically linked ICU is to have a library of a known version, so that behavior is determined and not dependent on any system libraries. You will most likely prefer to use the system ICU instead, as it may be updated over time without the need to recompile the Manticore daemon. In this case, you need to explicitly disable this option. This will also save you some space occupied by the ICU data file (about 30M), as it will not be included in the distribution.
• WITH_SSL - Used for support for HTTPS, and also encrypted MySQL connections to the daemon. The system OpenSSL library will be linked to the daemon. This implies that OpenSSL will be required to start the daemon. This is mandatory for support for HTTPS, but not strictly mandatory for the server (i.e. no SSL means no possibility to connect via HTTPS, but other protocols will work). SSL library versions starting from 1.0.2 to 1.1.1 may be used by Manticore, however note that for the sake of security it's highly recommended to use the freshest possible SSL
  library. For now only v1.1.1 is supported, the rest are outdated (
  see openssl release strategy
• WITH_ZLIB - used by the indexer to work with compressed columns from MySQL. Used by the daemon to provide support for the compressed MySQL protocol.
• WITH_ODBC - used by the indexer to support indexing sources from ODBC providers (they're typically UnixODBC and iODBC). On MS Windows, ODBC is the proper way to work with MS SQL sources, so indexing of MSSQL also implies this flag.
• DL_ODBC - don't link with the ODBC library. If ODBC is linked, but not available, you can't start indexer tool even if you want to process something not related to ODBC. This option asks the indexer to load the library at runtime only when you want to deal with ODBC source.
• ODBC_LIB - name of the ODBC library file. The indexer will try to load that file when you want to process ODBC source. This option is written automatically from available ODBC shared library investigation. You can also override that name at runtime by providing the environment variable ODBC_LIB with the proper path to an alternative library before running the indexer.
• WITH_EXPAT - used by the indexer to support indexing xmlpipe sources.
• DL_EXPAT - don't link with the EXPAT library. If EXPAT is linked, but not available, you can't start indexer tool even if you want to process something not related to xmlpipe. This option asks the indexer to load the library at runtime only when you want to deal with xmlpipe source.
• EXPAT_LIB - name of the EXPAT library file. The indexer will try to load that file when you want to process xmlpipe source. This option is written automatically from available EXPAT shared library investigation. You can also override that name at runtime by providing the environment variable EXPAT_LIB with the proper path to an alternative library before running the indexer.
• WITH_ICONV - for support different encodings when indexing xmlpipe sources with the indexer.
• DL_ICONV - don't link with the iconv library. If iconv is linked, but not available, you can't start indexer tool even if you want to process something not related to xmlpipe. This option asks the indexer to load the library at runtime only when you want to deal with xmlpipe source.
• ICONV_LIB - name of the iconv library file. The indexer will try to load that file when you want to process xmlpipe source. This option is written automatically from the available iconv shared library investigation. You can also override that name at runtime by providing the environment variable ICONV_LIB with the proper path to an alternative library before running the indexer.
• WITH_MYSQL - used by the indexer to support indexing MySQL sources.
• DL_MYSQL - don't link with the MySQL library. If MySQL is linked, but not available, you can't start theindexer tool even if you want to process something not related to MySQL. This option asks the indexer to load the library at runtime only when you want to deal with a MySQL source.
• MYSQL_LIB -- name of the MySQL library file. The indexer will try to load that file when you want to process a MySQL source. This option is written automatically from the available MySQL shared library investigation. You can also override that name at runtime by providing the environment variable MYSQL_LIB with the proper path to an alternative library before running the indexer.
• WITH_POSTGRESQL - used by the indexer to support indexing PostgreSQL sources.
• DL_POSTGRESQL - don't link with the PostgreSQL library. If PostgreSQL is linked, but not available, you can't start the indexer ool even if you want to process something not related to PostgreSQL. This option asks the indexer to load the library at runtime only when you want to deal with a PostgreSQL source.
• POSTGRESQL_LIB - name of postgresql library file. The indexer will attempt to load the specified postgresql library file when processing a postgresql source. This option is automatically determined from available postgresql shared library investigation. You can also override the name at runtime by providing the environment variable POSTGRESQL_LIB with the proper path to an alternative library before running the indexer.
• LOCALDATADIR - default path where the daemon stores binlogs. If this path is not provided or explicitly disabled in the daemon's runtime config (i.e. the file manticore.conf, which is not related to this build configuration), binlogs will be placed in this path. It is typically an absolute path, however, it is not required to be and relative paths can also be used. You probably would not need to change the default value defined by the configuration, which, depending on the target system, might be something like /var/data, /var/lib/manticore/data, or /usr/local/var/lib/manticore/data.
• FULL_SHARE_DIR - default path where all assets are stored. It can be overridden by the environment variable FULL_SHARE_DIR before starting any tool that utilizes files from that folder. This is an important path as many things are expected to be found there by default. These include predefined charset tables, stopwords, manticore modules, and icu data files, all placed in that folder. The configuration script usually determines this path to be something like /usr/share/manticore, or /usr/local/share/manticore.
• DISTR_BUILD - a shortcut for the options for releasing packages. This is a string value with the name of the target platform. It can be used instead of manually configuring all the options. On Debian and Redhat Linuxes, the default value might be determined by light introspection and set to a generic 'Debian' or 'RHEL'. Otherwise, the value is not defined.
• PACK - an even more convenient shortcut. It reads the DISTR environment variable, assigns it to the DISTR_BUILD parameter, and then works as usual. This is very useful when building in prepared build systems, like Docker containers, where the DISTR variable is set at the system level and reflects the target system for which the container is intended.
• CMAKE_INSTALL_PREFIX (path) - where Manticore is expected to be installed. Building does not perform any installations, but it prepares the installation rules that are executed when you run the cmake --install command or create a package and then install it. The prefix can be changed at any time, even during installation, by invoking
  cmake --install . --prefix /path/to/installation. However, at config time, this variable is used to initialize the default values of LOCALDATADIR and FULL_SHARE_DIR. For example, setting it to /my/custom at configure
  time will hardcode LOCALDATADIR as /my/custom/var/lib/manticore/data, and FULL_SHARE_DIR as
  /my/custom/usr/share/manticore.
• BUILD_TESTING (bool) whether to support testing. If enabled, after the build, you can run 'ctest' and test the build. Note that testing implies additional dependencies, like at least the presence of PHP cli, Python, and an available MySQL server with a test database. By default, this parameter is on. So, for 'just build', you might want to disable the option by explicitly specifying 'off' value.
• LIBS_BUNDLE - path to a folder with different libraries. This is mostly relevant for Windows building, but may also be helpful if you have to build often in order to avoid downloading third-party sources each time. By default, this path is never modified by the configuration script; you should put everything there manually. When, say, we want support for a stemmer - the sources will be downloaded from Snowball homepage, then extracted, configured, built, etc. Instead, you can store the original source tarball (which is libstemmer_c.tgz) in this folder. Next time you want to build from scratch, the configuration script will first look up in the bundle, and if it finds the stemmer there, it will not download it again from the Internet.
• CACHEB - path to a folder with stored builds of 3-rd party libraries. Usually features like galera, re2, icu, etc. first downloaded or being got from bundle, then unpacked, built, and installed into a temporary internal folder. When building manticore, that folder is then used as the place where the things required to support the asked feature are live. Finally, they either link with manticore, if it is a library; either go directly to distribution/installation (like galera or icu data). When CACHEB is defined either as cmake config param, either as a system environment variable, it is used as the target folder for that builds. This folder might be kept across builds, so that stored libraries there will not be rebuilt anymore, making the whole build process much shorter.

Note, that some options are organized in triples: WITH_XXX, DL_XXX and XXX_LIB - like support of mysql, odbc, etc. WITH_XXX determines whether next two have an effect or not. I.e., if you set WITH_ODBC to 0 - there is no sence to provide DL_ODBC and ODBC_LIB, and these two will have no effect if the whole feature is disabled. Also, XXX_LIB has no sense without DL_XXX, because if you don't want DL_XXX option, dynamic loading will not be used, and name provided by XXX_LIB is useless. That is used by default introspection.

Also, using the iconv library assumes expat and is useless if the last is disabled.

Also, some libraries may be always available, and so, there is no sense to avoid linkage with them. For example, in Windows that is ODBC. On macOS that is Expat, iconv, and m.b. others. Default introspection determines such libraries and effectively emits only WITH_XXX for them, without DL_XXX and XXX_LIB, that makes the things simpler.

With some options in game configuring might look like:

mkdir build && cd build
cmake -DWITH_MYSQL=1 -DWITH_RE2=1 ..

Apart general configuration values, you may also investigate file CMakeCache.txt which is left in build folder right after you run configuration. Any values defined there might be redefined explicitly when running cmake. For example, you may run cmake -DHAVE_GETADDRINFO_A=FALSE ..., and that config run will not assume investigated value of that variable, but will use one you've provided.

Specific environment variables

Environment variables are useful for providing some kind of global settings which are stored aside from build configuration and are always present. For persistence, they may be set globally on the system using different ways - like adding them to the .bashrc file, or embedding them into a Dockerfile if you produce a docker-based build system, or writing them in system preferences environment variables on Windows. Also, you may set them short-lived using export VAR=value in the shell. Or even shorter, by prepending values to the cmake call, like CACHEB=/my/cache cmake ... - this way it will only work on this call and will not be visible on the next.

Some of such variables are known to be used in general by cmake and some other tools. That is things like CXX which determines the current C++ compiler, or CXX_FLAGS to provide compiler flags, etc.

However, we have some variables that are specific to manticore configuration, which are invented solely for our builds.

• CACHEB - same as the config CACHEB option
• LIBS_BUNDLE - same as the config LIBS_BUNDLE option
• DISTR - used to initialize the DISTR_BUILD option when -DPACK=1 is used.
• DIAGNOSTIC - makes the output of cmake configuration much more verbose, explaining everything happening
• WRITEB - assumes LIBS_BUNDLE and, if set, will download source archive files for different tools to the LIBS_BUNDLE folder. That is, if a fresh version of the stemmer comes out - you can manually remove libstemmer_c.tgz from the bundle and then run a one-shot WRITEB=1 cmake ... - it will not find the stemmer's sources in the bundle and will then download them from the vendor's site to the bundle (without WRITEB it will download them into a temporary folder inside the build and will disappear when you wipe the build folder).

At the end of configuration, you may see what is available and will be used in a list like this one:

-- Enabled features compiled in:

* Galera, replication of tables
* re2, a regular expression library
* stemmer, stemming library (Snowball)
* icu, International Components for Unicode
* OpenSSL, for encrypted networking
* ZLIB, for compressed data and networking
* ODBC, for indexing MSSQL (windows) and generic ODBC sources with indexer
* EXPAT, for indexing xmlpipe sources with indexer
* Iconv, for support of different encodings when indexing xmlpipe sources with indexer
* MySQL, for indexing MySQL sources with indexer
* PostgreSQL, for indexing PostgreSQL sources with indexer

Building

cmake --build . --config RelWithDebInfo

Installation

To install run:

cmake --install . --config RelWithDebInfo

to install into custom (non-default) folder, run

cmake --install . --prefix path/to/build --config RelWithDebInfo

Building packages

For building a package, use the target package. It will build the package according to the selection provided by the -DDISTR_BUILD option. By default, it will be a simple .zip or .tgz archive with all binaries and supplementary files.

cmake --build . --target package --config RelWithDebInfo

Some advanced things about building

Recompilation (update) on single-config

If you haven't changed the path for sources and build, simply move to your build folder and run:

cmake .
cmake --build . --clean-first --config RelWithDebInfo

If by any reason it doesn't work, you can delete file CMakeCache.txt located in the build folder. After this step you
have to run cmake again, pointing to the source folder and configuring the options.

If it also doesn't help, just wipe out your build folder and begin from scratch.

Build types

Briefly - just use --config RelWithDebInfo as written above. It will make no mistake.

We use two build types. For development, it is Debug - it assigns compiler flags for optimization and other things in a way that it is very friendly for development, meaning the debug runs with step-by-step execution. However, the produced binaries are quite large and slow for production.

For releasing, we use another type - RelWithDebInfo - which means 'release build with debug info'. It produces production binaries with embedded debug info. The latter is then split away into separate debuginfo packages which are stored aside with release packages and might be used in case of some issues like crashes - for investigation and bugfixing. Cmake also provides Release and MinSizeRel, but we don't use them. If the build type is not available, cmake will make a noconfig build.

Build system generators

There are two types of generators: single-config and multi-config.

• Single-config needs the build type provided during configuration, via the CMAKE_BUILD_TYPE parameter. If it is not defined, the build will fall back to the RelWithDebInfo type which is suitable if you just want to build Manticore from sources and not participate in development. For explicit builds, you should provide a build type, like -DCMAKE_BUILD_TYPE=Debug.
• Multi-config selects the build type during the build. It should be provided with the --config option, otherwise it will build a kind of noconfig, which is not desirable. So, you should always specify the build type, like --config Debug.

If you want to specify the build type but don't want to care about whether it is a 'single' or 'multi' config generator - just provide the necessary keys in both places. I.e., configure with -DCMAKE_BUILD_TYPE=Debug, and then build with --config Debug. Just be sure that both values are the same. If the target builder is a single-config, it will consume the configuration param. If it is multi-config, the configuration param will be ignored, but the correct build configuration will be selected by the --config key.

If you want RelWithDebInfo (i.e. just build for production) and know you're on a single-config platform (that is all, except Windows) - you can omit the --config flag on the cmake invocation. The default CMAKE_BUILD_TYPE=RelWithDebInfo will be configured then, and used. All the commands for 'building', 'installation' and 'building package' will become shorter then.

Explicitly select build system generators

Cmake is the tool that doesn't perform building by itself, but it generates rules for the local build system.
Usually, it determines the available build system well, but sometimes you might need to provide a generator explicitly. You
can run cmake -G and review the list of available generators.

• On Windows, if you have more than one version of Visual Studio installed, you might need to specify which one to use,
  as:

cmake -G "Visual Studio 16 2019" ....
  ```
- On all other platforms - usually Unix Makefiles are used, but you can specify another one, such as Ninja, or Ninja Multi-Config, as:
  Multi-Config`, as:
```bash
  cmake -GNinja ...
  ```
  or
```bash
  cmake -G"Ninja Multi-Config" ...

Ninja Multi-Config is quite useful as it is really 'multi-config' and available on Linux/macOS/BSD. With this generator, you may shift the choosing of configuration type to build time, and also you may build several configurations in one and the same build folder, changing only the --config param.

Caveats

1. If you want to finally build a full-featured RPM package, the path to the build directory must be long enough in order to correctly build debug symbols.
   Like /manticore012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789, for example. That is because RPM tools modify the path over compiled binaries when building debug info, and it can just write over existing room and won't allocate more. The aforementioned long path has 100 chars and that is quite enough for such a case.

External dependencies

Some libraries should be available if you want to use them.
- For indexing (indexer tool): expat, iconv, mysql, odbc, postgresql. Without them, you can only process tsv and csv sources.
- For serving queries (searchd daemon): openssl might be necessary.
- For all (required, mandatory!) we need the Boost library. The minimal version is 1.61.0, however, we build the binaries with a fresher version 1.75.0. Even more recent versions (like 1.76) should also be okay. On Windows, you can download pre-built Boost from their site (boost.org) and install it into the default suggested path (i.e. C:\\boost...). On MacOs, the one provided in brew is okay. On Linux, you can check the available version in official repositories, and if it doesn't match requirements, you can build from sources. We need the component 'context', you can also build components 'system' and 'program_options', they will be necessary if you also want to build Galera library from the sources. Look into dist/build_dockers/xxx/boost_175/Dockerfile for a short self-documented script/instruction on how to do it.

On the build system, you need the 'dev' or 'devel' versions of these packages installed (i.e. - libmysqlclient-devel, unixodbc-devel, etc. Look to our dockerfiles for the names of concrete packages).

On run systems, these packages should be present at least in the final (non-dev) variants. (devel variants usually larger, as they include not only target binaries, but also different development stuff like include headers, etc.).

Building on Windows

Apart from necessary prerequisites, you might need prebuilt expat, iconv, mysql, and postgresql client libraries. You have to either build them yourself or contact us to get our build bundle (a simple zip archive where the folder with these targets is located).

• ODBC is not necessary as it is a system library.
• OpenSSL might be built from sources or downloaded prebuilt from https://slproweb.com/products/Win32OpenSSL.html (as mentioned in the cmake internal script on FindOpenSSL).
• Boost might be downloaded pre-built from https://www.boost.org/ releases.

See what is compiled

Run indexer -h. It will show which features were configured and built (whether they're explicit or investigated, doesn't matter):

Built on Linux x86_64 by GNU 8.3.1 compiler.

Configured with these definitions: -DDISTR_BUILD=rhel8 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1
-DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ODBC=1 -DDL_ODBC=1
-DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1
-DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data
-DFULL_SHARE_DIR=/usr/share/manticore

Sphinx 2.x -> Manticore 2.x

Manticore Search 2.x maintains compatibility with Sphinxsearch 2.x and can load existing tables created by Sphinxsearch. In most cases, upgrading is just a matter of replacing the binaries.

Instead of sphinx.conf (in Linux normally located at /etc/sphinxsearch/sphinx.conf) Manticore by default uses /etc/manticoresearch/manticore.conf. It also runs under a different user and use different folders.

Systemd service name has changed from sphinx/sphinxsearch to manticore and the service runs under user manticore (Sphinx was using sphinx or sphinxsearch). It also uses a different folder for the PID file.

The folders used by default are /var/lib/manticore, /var/log/manticore, /var/run/manticore. You can still use the existing Sphinx config, but you need to manually change permissions for /var/lib/sphinxsearch and /var/log/sphinxsearch folders. Or, just rename globally 'sphinx' to 'manticore' in system files. If you use other folders (for data, wordforms files etc.) the ownership must be also switched to user manticore. The pid_file location should be changed to match the manticore.service to /var/run/manticore/searchd.pid.

If you want to use the Manticore folder instead, the table files need to be moved to the new data folder (/var/lib/manticore) and the permissions must be changed to user manticore.

Sphinx 2.x / Manticore 2.x -> Manticore 3.x

Upgrading from Sphinx / Manticore 2.x to 3.x is not straightforward, as the table storage engine has undergone a significant upgrade and the new searchd cannot load older tables and upgrade them to the new format on-the-fly.

Manticore Search 3 got a redesigned table storage. Tables created with Manticore/Sphinx 2.x cannot be loaded by Manticore Search 3 without a conversion. Because of the 4GB limitation, a real-time table in 2.x could still have several disk chunks after an optimize operation. After upgrading to 3.x, these tables can now be optimized to 1-disk chunk with the usual OPTIMIZE command. Index files also changed. The only component that didn't get any structural changes is the .spp file (hitlists). .sps (strings/json) and .spm (MVA) are now held by .spb (var-length attributes). The new format has an .spm file present, but it's used for row map (previously it was dedicated for MVA attributes). The new extensions added are .spt (docid lookup), .sphi ( secondary index histograms), .spds (document storage). In case you are using scripts that manipulate table files, they should be adapted for the new file extensions.

The upgrade procedure may differ depending on your setup (number of servers in the cluster, whether you have high availability or not, etc.), but in general, it involves creating new 3.x table versions and replacing your existing ones, as well as replacing older 2.x binaries with the new ones.

There are two special requirements to take care:

• Real-time tables need to be flushed using FLUSH RAMCHUNK
• Plain tables with kill-lists require adding a new directive in table configuration (see killlist_target)

Manticore Search 3 includes a new tool - index_converter - that can convert Sphinx 2.x / Manticore 2.x tables to 3.x format. index_converter comes in a separate package which should be installed first. Using the convert tool create 3.x versions of your tables. index_converter can write the new files in the existing data folder and backup the old files or it can write the new files to a chosen folder.

Basic upgrade instruction

If you have a single server:

• Install the manticore-converter package
• Use index_converter to create new versions of the tables in a different folder than the existing data folder (using the --output-dir option)
• Stop the existing Manticore/Sphinx, upgrade to 3.0, move the new tables to the data folder, and start Manticore

To minimize downtime, you can copy 2.x tables, config (you'll need to edit paths here for tables, logs, and different ports), and binaries to a separate location and start this on a separate port. Point your application to it. After upgrading to 3.0 and the new server is started, you can point the application back to the normal ports. If everything is good, stop the 2.x copy and delete the files to free up space.

If you have a spare box (like a testing or staging server), you can do the table upgrade there first and even install Manticore 3 to perform several tests. If everything is okay, copy the new table files to the production server. If you have multiple servers that can be pulled out of production, do it one by one and perform the upgrade on each. For distributed setups, 2.x searchd can work as a master with 3.x nodes, so you can do the upgrading on the data nodes first, and then on the master node.

There have been no changes made to the way clients should connect to the engine, or any changes to the querying mode or behavior of queries.

kill-lists in Sphinx / Manticore 2.x vs Manticore 3.x

Kill-lists have been redesigned in Manticore Search 3. In previous versions, kill-lists were applied to the result set provided by each previously searched table at query time.

Thus, in 2.x, the table order at query time mattered. For example, if a delta table had a kill-list, in order to apply it against the main table, the order had to be main, delta (either in a distributed table or in the FROM clause).

In Manticore 3, kill-lists are applied to a table when it's loaded during searchd startup or gets rotated. The new directive killlist_target in table configuration specifies target tables and defines which doc ids from the source table should be used for suppression. These can be ids from the defined kill-list, actual doc ids of the table or both.

Documents from the kill-lists are deleted from the target tables, they are not returned in results even if the search doesn't include the table that provided the kill-lists. Because of that, the order of tables for searching does not matter anymore. Now, delta, main and main, delta will provide the same results.

In previous versions, tables were rotated following the order from the configuration file. In Manticore 3 table rotation order is much smarter and works in accordance with killlist targets. Before starting to rotate tables, the server looks for chains of tables by killlist_target definitions. It will then first rotate tables not referenced anywhere as kill-lists targets. Next, it will rotate tables targeted by already rotated tables and so on. For example, if we do indexer --all and we have 3 tables: main, delta_big (which targets at the main) and delta_small (with target at delta_big), first, delta_small is rotated, then delta_big and finally the main. This is to ensure that when a dependent table is rotated it gets the most actual kill-list from other tables.

Configuration keys removed in Manticore 3.x

• docinfo - everything is now extern
• inplace_docinfo_gap - not needed anymore
• mva_updates_pool - MVAs don’t have anymore a dedicated pool for updates, as now they can be updated directly in the blob (see below).

Updating var-length attributes in Manticore 3.x

String, JSON and MVA attributes can be updated in Manticore 3.x using UPDATE statement.

In 2.x string attributes required REPLACE, for JSON it was only possible to update scalar properties (as they were fixed-width) and MVAs could be updated using the MVA pool. Now updates are performed directly on the blob component. One setting that may require tuning is attr_update_reserve which allows changing the allocated extra space at the end of the blob used to avoid frequent resizes in case the new values are bigger than the existing values in the blob.

Document IDs in Manticore 3.x

Doc ids used to be UNSIGNED 64-bit integers. Now they are POSITIVE SIGNED 64-bit integers.

RT mode in Manticore 3.x

Read here about the RT mode

Special suffixes since Manticore 3.x

Manticore 3.x recognizes and parses special suffixes which makes easier to use numeric values with special meaning. Common form for them is integer number + literal, like 10k or 100d, but not 40.3s (since 40.3 is not integer), or not 2d 4h (since there are two, not one value). Literals are case-insensitive, so 10W is the same as 10w. There are 2 types of such suffixes currently supported:

• Size suffixes - can be used in parameters that define size of something (memory buffer, disk file, limit of RAM, etc. ) in bytes. "Naked" numbers in that places mean literally size in bytes (octets). Size values take suffix k for kilobytes (1k=1024), m for megabytes (1m=1024k), g for gigabytes (1g=1024m) and t for terabytes (1t=1024g).
• Time suffixes - can be used in parameters defining some time interval values like delays, timeouts, etc. "Naked" values for those parameters usually have documented scale, and you must know if their numbers, say, 100, means '100 seconds' or '100 milliseconds'. However instead of guessing you just can write suffixed value and it will be fully determined by it's suffix. Time values take suffix us for useconds (microseconds), ms for milliseconds, s for seconds, m for minutes, h for hours, d for days and w for weeks.

index_converter

index_converter is a tool for converting tables created with Sphinx/Manticore Search 2.x to the Manticore Search 3.x table format. The tool can be used in several different ways:

Convert one table at a time

$ index_converter --config /home/myuser/manticore.conf --index tablename

Convert all tables

$ index_converter --config /home/myuser/manticore.conf --all

Convert tables found in a folder

$ index_converter  --path /var/lib/manticoresearch/data --all

The new version of the table is written by default in the same folder. The previous version's files are saved with the .old extension in their name. An exception is the .spp (hitlists) file, which is the only table component that didn't have any changes in the new format.

You can save the new table version to a different folder using the -–output-dir option

$ index_converter --config /home/myuser/manticore.conf --all --output-dir /new/path

Convert kill lists

A special case is for tables containing kill-lists. As the behaviour of how kill-lists works has changed (see killlist_target), the delta table should know which are the target tables for applying the kill-lists. There are 3 ways to have a converted table ready for setting targeted tables for applying kill-lists:

• Use -–killlist-target when converting a table
  ini $ index_converter --config /home/myuser/manticore.conf --index deltaindex --killlist-target mainindex:kl

• Add killlist_target in the configuration before doing the conversion

• use ALTER ... KILLIST_TARGET command after conversion

Complete list of index_converter options

Here's the complete list of index_converter options:

• --config <file> (-c <file> for short) tells index_converter to use the given file as its configuration. Normally, it will look for manticore.conf in the installation directory (e.g. /usr/local/manticore/etc/manticore.conf if installed into /usr/local/sphinx), followed by the current directory you are in when calling index_converter from the shell.
• --index specifies which table should be converted
• --path - instead of using a config file, a path containing table(s) can be used
• --strip-path - strips path from filenames referenced by table: stopwords, exceptions and wordforms
• --large-docid - allows to convert documents with ids larger than 2^63 and display a warning, otherwise it will just exit on the large id with an error. This option was added as in Manticore 3.x doc ids are signed bigint, while previously they were unsigned
• --output-dir <dir> - writes the new files in a chosen folder rather than the same location as with the existing table files. When this option set, existing table files will remain untouched at their location.
• --all - converts all tables from the config
• --killlist-target <targets> sets the target tables for which kill-lists will be applied. This option should be used only in conjunction with the --index option

Install and start Manticore

You can install and start Manticore easily on various operating systems, including Ubuntu, Centos, Debian, Windows, and MacOS. Additionally, you can also use Manticore as a Docker container.

Ubuntu

wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore manticore-columnar-lib
sudo systemctl start manticore

wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore manticore-columnar-lib
sudo systemctl start manticore

sudo yum install https://repo.manticoresearch.com/manticore-repo.noarch.rpm
sudo yum install manticore manticore-columnar-lib
sudo systemctl start manticore

brew install manticoresearch
brew services start manticoresearch

docker pull manticoresearch/manticore
docker run -e EXTRA=1 --name manticore -p9306:9306 -p9308:9308 -p9312:9312 -d manticoresearch/manticore

Connect to Manticore

By default Manticore is waiting for your connections on:

• port 9306 for MySQL clients
• port 9308 for HTTP/HTTPS connections
• port 9312 for connections from other Manticore nodes and clients based on Manticore binary API

More details about HTTPS support can be found in our learning course here.

mysql -h0 -P9306

curl -s "http://localhost:9308/search"

// https://github.com/manticoresoftware/manticoresearch-php
require_once __DIR__ . '/vendor/autoload.php';
$config = ['host'=>'127.0.0.1','port'=>9308];
$client = new \Manticoresearch\Client($config);

// https://github.com/manticoresoftware/manticoresearch-python
import manticoresearch
config = manticoresearch.Configuration(
    host = "http://127.0.0.1:9308"
)
client = manticoresearch.ApiClient(config)
indexApi = manticoresearch.IndexApi(client)
searchApi = manticoresearch.SearchApi(client)
utilsApi = manticoresearch.UtilsApi(client)

// https://github.com/manticoresoftware/manticoresearch-javascript
var Manticoresearch = require('manticoresearch');
var client= new Manticoresearch.ApiClient()
client.basePath="http://127.0.0.1:9308";
indexApi = new Manticoresearch.IndexApi(client);
searchApi = new Manticoresearch.SearchApi(client);
utilsApi = new Manticoresearch.UtilsApi(client);

// https://github.com/manticoresoftware/manticoresearch-java
import com.manticoresearch.client.*;
import com.manticoresearch.client.model.*;
import com.manticoresearch.client.api.*;
...
ApiClient client = Configuration.getDefaultApiClient();
client.setBasePath("http://127.0.0.1:9308");
...
IndexApi indexApi = new IndexApi(client);
SearchApi searchApi = new UtilsApi(client);
UtilsApi utilsApi = new UtilsApi(client);

// https://github.com/manticoresoftware/manticoresearch-net
using System.Net.Http;
...
using ManticoreSearch.Client;
using ManticoreSearch.Api;
using ManticoreSearch.Model;
...
config = new Configuration();
config.BasePath = "http://localhost:9308";
httpClient = new HttpClient();
httpClientHandler = new HttpClientHandler();
...
var indexApi = new IndexApi(httpClient, config, httpClientHandler);
var searchApi = new SearchApi(httpClient, config, httpClientHandler);
var utilsApi = new UtilsApi(httpClient, config, httpClientHandler);

import {
  Configuration,
  IndexApi,
  SearchApi,
  UtilsApi
} from "manticoresearch-ts";
...
const config = new Configuration({
  basePath: 'http://localhost:9308',
})
const indexApi = new IndexApi(config);
const searchApi = new SearchApi(config);
const utilsApi = new UtilsApi(config);

import (
    "context"
    manticoreclient "github.com/manticoresoftware/manticoresearch-go"
)
...
configuration := manticoreclient.NewConfiguration()
configuration.Servers[0].URL = "http://localhost:9308"
apiClient := manticoreclient.NewAPIClient(configuration)

Create a table

Let's now create a table called "products" with 2 fields:

• title - full-text field which will contain our product's title
• price - of type "float"

Note that it is possible to omit creating a table with an explicit create statement. For more information, see Auto schema.

More information about different ways to create a table can be found in our learning courses:

• Creating a RealTime table
• Creating a table from the MySQL source
• Creating a table from the CSV source
• Creating a table using the auto schema mechanism
• Creating a table with Logstash/Beats
• Creating a table with Fluentbit
• Creating a table using the Vector.dev agent

create table products(title text, price float) morphology='stem_en';

Query OK, 0 rows affected (0.02 sec)

POST /cli -d "create table products(title text, price float) morphology='stem_en'"

{
"total":0,
"error":"",
"warning":""
}

$index = new \Manticoresearch\Index($client);
$index->setName('products');
$index->create([
    'title'=>['type'=>'text'],
    'price'=>['type'=>'float'],
],['morphology' => 'stem_en']);

utilsApi.sql('create table products(title text, price float) morphology=\'stem_en\'')

res = await utilsApi.sql('create table products(title text, price float) morphology=\'stem_en\'');

utilsApi.sql("create table products(title text, price float) morphology='stem_en'");

utilsApi.Sql("create table products(title text, price float) morphology='stem_en'");

res = await utilsApi.sql('create table products(title text, price float) morphology=\'stem_en\'');

res := apiClient.UtilsAPI.Sql(context.Background()).Body("create table products(title text, price float) morphology='stem_en'").Execute();

Add documents

Let's now add few documents to the table:

insert into products(title,price) values ('Crossbody Bag with Tassel', 19.85), ('microfiber sheet set', 19.99), ('Pet Hair Remover Glove', 7.99);

Query OK, 3 rows affected (0.01 sec)

POST /insert
{
  "index":"products",
  "doc":
  {
    "title" : "Crossbody Bag with Tassel",
    "price" : 19.85
  }
}


POST /insert
{
  "index":"products",
  "doc":
  {
    "title" : "microfiber sheet set",
    "price" : 19.99
  }
}

POST /insert
{
  "index":"products",
  "doc":
  {
    "title" : "Pet Hair Remover Glove",
    "price" : 7.99
  }
}

{
  "_index": "products",
  "_id": 0,
  "created": true,
  "result": "created",
  "status": 201
}

{
  "_index": "products",
  "_id": 0,
  "created": true,
  "result": "created",
  "status": 201
}

{
  "_index": "products",
  "_id": 0,
  "created": true,
  "result": "created",
  "status": 201
}

$index->addDocuments([
        ['title' => 'Crossbody Bag with Tassel', 'price' => 19.85],
        ['title' => 'microfiber sheet set', 'price' => 19.99],
        ['title' => 'Pet Hair Remover Glove', 'price' => 7.99]
]);

indexApi.insert({"index" : "products", "doc" : {"title" : "Crossbody Bag with Tassel", "price" : 19.85}})
indexApi.insert({"index" : "products", "doc" : {"title" : "microfiber sheet set", "price" : 19.99}})
indexApi.insert({"index" : "products", "doc" : {"title" : "Pet Hair Remover Glove", "price" : 7.99}})

res = await indexApi.insert({"index" : "products", "doc" : {"title" : "Crossbody Bag with Tassel", "price" : 19.85}});
res = await indexApi.insert({"index" : "products", "doc" : {"title" : "microfiber sheet set", "price" : 19.99}});
res = await indexApi.insert({"index" : "products", "doc" : {"title" : "Pet Hair Remover Glove", "price" : 7.99}});

InsertDocumentRequest newdoc = new InsertDocumentRequest();
HashMap<String,Object> doc = new HashMap<String,Object>(){{
    put("title","Crossbody Bag with Tassel");
    put("price",19.85);
}};
newdoc.index("products").setDoc(doc);
sqlresult = indexApi.insert(newdoc);

newdoc = new InsertDocumentRequest();
doc = new HashMap<String,Object>(){{
    put("title","microfiber sheet set");
    put("price",19.99);
}};
newdoc.index("products").setDoc(doc);
sqlresult = indexApi.insert(newdoc);

newdoc = new InsertDocumentRequest();
doc = new HashMap<String,Object>(){{
    put("title","Pet Hair Remover Glove");
    put("price",7.99);
 }};
newdoc.index("products").setDoc(doc);
indexApi.insert(newdoc);

Dictionary<string, Object> doc = new Dictionary<string, Object>();
doc.Add("title","Crossbody Bag with Tassel");
doc.Add("price",19.85);
InsertDocumentRequest insertDocumentRequest = new InsertDocumentRequest(index: "products", doc: doc);
sqlresult = indexApi.Insert(insertDocumentRequest);

doc = new Dictionary<string, Object>();
doc.Add("title","microfiber sheet set");
doc.Add("price",19.99);
insertDocumentRequest = new InsertDocumentRequest(index: "products", doc: doc);
sqlresult = indexApi.Insert(insertDocumentRequest);

doc = new Dictionary<string, Object>();
doc.Add("title","Pet Hair Remover Glove");
doc.Add("price",7.99);
insertDocumentRequest = new InsertDocumentRequest(index: "products", doc: doc);
sqlresult = indexApi.Insert(insertDocumentRequest);

res = await indexApi.insert({
  index: 'test',
  id: 1,
  doc: { content: 'Text 1', name: 'Doc 1', cat: 1 },
});
res = await indexApi.insert({
  index: 'test',
  id: 2,
  doc: { content: 'Text 2', name: 'Doc 2', cat: 2 },
});
res = await indexApi.insert({
  index: 'test',
  id: 3,
  doc: { content: 'Text 3', name: 'Doc 3', cat: 7 },
});

indexDoc := map[string]interface{} {"content": "Text 1", "name": "Doc 1", "cat": 1 }
indexReq := manticoreclient.NewInsertDocumentRequest("products", indexDoc)
indexReq.SetId(1)
apiClient.IndexAPI.Insert(context.Background()).InsertDocumentRequest(*indexReq).Execute()

indexDoc = map[string]interface{} {"content": "Text 2", "name": "Doc 3", "cat": 2 }
indexReq = manticoreclient.NewInsertDocumentRequest("products", indexDoc)
indexReq.SetId(2)
apiClient.IndexAPI.Insert(context.Background()).InsertDocumentRequest(*indexReq).Execute()

indexDoc = map[string]interface{} {"content": "Text 3", "name": "Doc 3", "cat": 7 }     
indexReq = manticoreclient.NewInsertDocumentRequest("products", indexDoc)
indexReq.SetId(3)
apiClient.IndexAPI.Insert(context.Background()).InsertDocumentRequest(*indexReq).Execute()

More details on the subject can be found here:

• Adding data to a plain table
• Adding data to a RealTime table

Search

Let's find one of the documents. The query we will use is 'remove hair'. As you can see, it finds a document with the title 'Pet Hair Remover Glove' and highlights 'Hair remover' in it, even though the query has "remove", not "remover". This is because when we created the table, we turned on using English stemming (morphology "stem_en").

select id, highlight(), price from products where match('remove hair');

+---------------------+-------------------------------+----------+
| id                  | highlight()                   | price    |
+---------------------+-------------------------------+----------+
| 1513686608316989452 | Pet <b>Hair Remover</b> Glove | 7.990000 |
+---------------------+-------------------------------+----------+
1 row in set (0.00 sec)

POST /search
{
  "index": "products",
  "query": { "match": { "title": "remove hair" } },
  "highlight":
  {
    "fields": ["title"]
  }
}

{
  "took": 0,
  "timed_out": false,
  "hits": {
    "total": 1,
    "hits": [
      {
        "_id": "1513686608316989452",
        "_score": 1680,
        "_source": {
          "price": 7.99,
          "title": "Pet Hair Remover Glove"
        },
        "highlight": {
          "title": [
            "Pet <b>Hair Remover</b> Glove"
          ]
        }
      }
    ]
  }
}

$result = $index->search('@title remove hair')->highlight(['title'])->get();
foreach($result as $doc)
{
    echo "Doc ID: ".$doc->getId()."\n";
    echo "Doc Score: ".$doc->getScore()."\n";
    echo "Document fields:\n";
    print_r($doc->getData());
    echo "Highlights: \n";
    print_r($doc->getHighlight());
}

Doc ID: 1513686608316989452
Doc Score: 1680
Document fields:
Array
(
    [price] => 7.99
    [title] => Pet Hair Remover Glove
)
Highlights:
Array
(
    [title] => Array
        (
            [0] => Pet <b>Hair Remover</b> Glove
        )
)

searchApi.search({"index":"products","query":{"query_string":"@title remove hair"},"highlight":{"fields":["title"]}})

{'hits': {'hits': [{u'_id': u'1513686608316989452',
                    u'_score': 1680,
                    u'_source': {u'title': u'Pet Hair Remover Glove', u'price':7.99},
                    u'highlight':{u'title':[u'Pet <b>Hair Remover</b> Glove']}}}],
          'total': 1},
 'profile': None,
 'timed_out': False,
 'took': 0}

res = await searchApi.search({"index":"products","query":{"query_string":"@title remove hair"}"highlight":{"fields":["title"]}});

{"hits": {"hits": [{"_id": "1513686608316989452",
                    "_score": 1680,
                    "_source": {"title": "Pet Hair Remover Glove", "price":7.99},
                    "highlight":{"title":["Pet <b>Hair Remover</b> Glove"]}}],
          "total": 1},
 "profile": None,
 "timed_out": False,
 "took": 0}

query = new HashMap<String,Object>();
query.put("query_string","@title remove hair");
searchRequest = new SearchRequest();
searchRequest.setIndex("forum");
searchRequest.setQuery(query);
HashMap<String,Object> highlight = new HashMap<String,Object>(){{
    put("fields",new String[] {"title"});

}};
searchRequest.setHighlight(highlight);
searchResponse = searchApi.search(searchRequest);

class SearchResponse {
    took: 84
    timedOut: false
    hits: class SearchResponseHits {
        total: 1
        maxScore: null
        hits: [{_id=1513686608316989452, _score=1, _source={price=7.99, title=Pet Hair Remover Glove}, highlight={title=[Pet <b>Hair Remover</b> Glove]}}]
        aggregations: null
    }
    profile: null
}

object query =  new { query_string="@title remove hair" };
var searchRequest = new SearchRequest("products", query);
var highlight = new Highlight();
highlight.Fieldnames = new List<string> {"title"};
searchRequest.Highlight = highlight;
searchResponse = searchApi.Search(searchRequest);

class SearchResponse {
    took: 103
    timedOut: false
    hits: class SearchResponseHits {
        total: 1
        maxScore: null
        hits: [{_id=1513686608316989452, _score=1, _source={price=7.99, title=Pet Hair Remover Glove}, highlight={title=[Pet <b>Hair Remover</b> Glove]}}]
        aggregations: null
    }
    profile: null
}

res = await searchApi.search({
  index: 'test',
  query: { query_string: {'text 1'} },
  highlight: {'fields': ['content'] }
});

{
    "hits": 
    {
        "hits": 
        [{
            "_id": "1",
            "_score": 1400,
            "_source": {"content":"Text 1","name":"Doc 1","cat":1},
            "highlight": {"content":["<b>Text 1</b>"]}
        }],
        "total": 1
    },
    "profile": None,
    "timed_out": False,
    "took": 0
}

searchRequest := manticoreclient.NewSearchRequest("test")
query := map[string]interface{} {"query_string": "text 1"};
searchRequest.SetQuery(query);

highlightField := manticoreclient.NewHighlightField("content")
fields := []interface{}{ highlightField }
highlight := manticoreclient.NewHighlight()
highlight.SetFields(fields)
searchRequest.SetHighlight(highlight);

res, _, _ := apiClient.SearchAPI.Search(context.Background()).SearchRequest(*searchRequest).Execute()

{
    "hits": 
    {
        "hits": 
        [{
            "_id": "1",
            "_score": 1400,
            "_source": {"content":"Text 1","name":"Doc 1","cat":1},
            "highlight": {"content":["<b>Text 1</b>"]}
        }],
        "total": 1
    },
    "profile": None,
    "timed_out": False,
    "took": 0
}

More information on different search options available in Manticore can be found in our learning courses:

• Faceted search
• Geo search
• Searching for similar documents

Update

Let's assume we now want to update the document - change the price to 18.5. This can be done by filtering by any field, but normally you know the document id and update something based on that.

update products set price=18.5 where id = 1513686608316989452;

Query OK, 1 row affected (0.00 sec)

POST /update
{
  "index": "products",
  "id": 1513686608316989452,
  "doc":
  {
    "price": 18.5
  }
}

{
  "_index": "products",
  "_id": 1513686608316989452,
  "result": "updated"
}

$doc = [
    'body' => [
        'index' => 'products',
        'id' => 2,
        'doc' => [
            'price' => 18.5
        ]
    ]
];

$response = $client->update($doc);

indexApi = api = manticoresearch.IndexApi(client)
indexApi.update({"index" : "products", "id" : 1513686608316989452, "doc" : {"price":18.5}})

res = await indexApi.update({"index" : "products", "id" : 1513686608316989452, "doc" : {"price":18.5}});

UpdateDocumentRequest updateRequest = new UpdateDocumentRequest();
doc = new HashMap<String,Object >(){{
    put("price",18.5);
}};
updateRequest.index("products").id(1513686608316989452L).setDoc(doc);
indexApi.update(updateRequest);

Dictionary<string, Object> doc = new Dictionary<string, Object>();
doc.Add("price", 18.5);
UpdateDocumentRequest updateDocumentRequest = new UpdateDocumentRequest(index: "products", id: 1513686608316989452L, doc: doc);
indexApi.Update(updateDocumentRequest);

res = await indexApi.update({ index: "test", id: 1, doc: { cat: 10 } });

updDoc = map[string]interface{} {"cat": 10}
updRequest = manticoreclient.NewUpdateDocumentRequest("test", updDoc)
updRequest.SetId(1)
res, _, _ = apiClient.IndexAPI.Update(context.Background()).UpdateDocumentRequest(*updRequest).Execute()

Delete

Let's now delete all documents with price lower than 10.

delete from products where price < 10;

Query OK, 1 row affected (0.00 sec)

POST /delete
{
  "index": "products",
  "query":
  {
    "range":
    {
      "price":
      {
        "lte": 10
      }
    }
  }
}

{
  "_index": "products",
  "deleted": 1
}

$result = $index->deleteDocuments(new \Manticoresearch\Query\Range('price',['lte'=>10]));

Array
(
    [_index] => products
    [deleted] => 1
)

indexApi.delete({"index" : "products", "query": {"range":{"price":{"lte":10}}}})

res = await indexApi.delete({"index" : "products", "query": {"range":{"price":{"lte":10}}}});

DeleteDocumentRequest deleteRequest = new DeleteDocumentRequest();
query = new HashMap<String,Object>();
query.put("range",new HashMap<String,Object>(){{
    put("price",new HashMap<String,Object>(){{
        put("lte",10);
    }});
}});
deleteRequest.index("products").setQuery(query);
indexApi.delete(deleteRequest);

Dictionary<string, Object> price = new Dictionary<string, Object>();
price.Add("lte", 10);
Dictionary<string, Object> range = new Dictionary<string, Object>();
range.Add("price", price);
DeleteDocumentRequest deleteDocumentRequest = new DeleteDocumentRequest(index: "products", range: range);
indexApi.Delete(deleteDocumentRequest);

res = await indexApi.delete({
  index: 'test',
  query: { match: { '*': 'Text 1' } },
});

delRequest := manticoreclient.NewDeleteDocumentRequest("test")
matchExpr := map[string]interface{} {"*": "Text 1t"}
delQuery := map[string]interface{} {"match": matchExpr }
delRequest.SetQuery(delQuery)
res, _, _ := apiClient.IndexAPI.Delete(context.Background()).DeleteDocumentRequest(*delRequest).Execute();

Manticore Search server can be started using different methods, depending on the installation type.

Starting and stopping using systemd

After the installation the Manticore Search service is not started automatically. To start Manticore run the following command:

sudo systemctl start manticore

To stop Manticore run the following command:

sudo systemctl stop manticore

The Manticore service is set to run at boot. You can check it by running:

sudo systemctl is-enabled manticore

If you want to disable Manticore from starting at boot time, run:

sudo systemctl disable manticore

To make Manticore start at boot, run:

sudo systemctl enable manticore

searchd process logs startup information in systemd journal. If systemd logging is enabled you can view the logged information with the following command:

sudo journalctl -u manticore

Custom startup flags using systemd

systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS allows you to specify custom startup flags that the Manticore Search daemon should be started with. See full list here.

For example, to start Manticore with the debug logging level, you can run:

systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS='--logdebug'
systemctl restart manticore

To undo it, run:

systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS=''
systemctl restart manticore

Note, systemd environment variables get reset on server reboot.

Starting and stopping using service

Manticore can be started and stopped using service commands:

sudo service manticore start
sudo service manticore stop

To enable the sysV service at boot on RedHat systems run:

chkconfig manticore on

To enable the sysV service at boot on Debian systems (including Ubuntu) run:

update-rc.d manticore defaults

Please note that searchd is started by the init system under the manticore user and all files created by the server will be owned by this user. If searchd is started under, for example, the root user, the file permissions will be changed, which may result in issues when running searchd as a service again.

You can also start Manticore Search by calling searchd (Manticore Search server binary) directly:

searchd [OPTIONS]

Note that without specifying a path to the configuration file, searchd will try to find it in several locations depending on the operating system.

searchd command line options

The options available to searchd in all operating systems are:

• --help (-h for short) lists all of the parameters that can be used in your particular build of searchd.
• --version (-v for short) shows Manticore Search version information.
• --config <file> (-c <file> for short) tells searchd to use the specified file as its configuration.
• --stop is used to asynchronously stop searchd, using the details of the PID file as specified in the Manticore configuration file. Therefore, you may also need to confirm to searchd which configuration file to use with the --config option. Example:

bash $ searchd --config /etc/manticoresearch/manticore.conf --stop

• --stopwait is used to synchronously stop searchd. --stop essentially tells the running instance to exit (by sending it a SIGTERM) and then immediately returns. --stopwait will also attempt to wait until the running searchd instance actually finishes the shutdown (eg. saves all the pending attribute changes) and exits. Example:

bash $ searchd --config /etc/manticoresearch/manticore.conf --stopwait

Possible exit codes are as follows:

• 0 on success
• 1 if connection to running searchd server failed
• 2 if server reported an error during shutdown

• 3 if server crashed during shutdown

• --status command is used to query running searchd instance status using the connection details from the (optionally) provided configuration file. It will try to connect to running instance using the first found UNIX socket or TCP port from the configuration file. On success it will query for a number of status and performance counter values and print them. You can also use SHOW STATUS command to access the very same counters via SQL protocol. Examples:

bash $ searchd --status $ searchd --config /etc/manticoresearch/manticore.conf --status

• --pidfile is used to explicitly force using a PID file (where the searchd process identification number is stored) despite any other debugging options that say otherwise (for instance, --console). This is a debugging option.

bash $ searchd --console --pidfile

• --console is used to force searchd into console mode. Typically, Manticore runs as a conventional server application and logs information into log files (as specified in the configuration file). However, when debugging issues in the configuration or the server itself, or trying to diagnose hard-to-track-down problems, it may be easier to force it to dump information directly to the console/command line from which it is being called. Running in console mode also means that the process will not be forked (so searches are done in sequence) and logs will not be written to. (It should be noted that console mode is not the intended method for running searchd.) You can invoke it as:

bash $ searchd --config /etc/manticoresearch/manticore.conf --console

• --logdebug, --logreplication, --logdebugv, and --logdebugvv options enable additional debug output in the server log. They differ by the logging verboseness level. These are debugging options and should not be normally enabled, as they can pollute the log a lot. They can be used temporarily on request to assist with complicated debugging sessions.

• --iostats is used in conjunction with the logging options (the query_log must have been activated in manticore.conf) to provide more detailed information on a per-query basis about the input/output operations carried out in the course of that query, with a slight performance hit and slightly bigger logs. The IO statistics don't include information about IO operations for attributes, as these are loaded with mmap. To enable it, you can start searchd as follows:

bash $ searchd --config /etc/manticoresearch/manticore.conf --iostats

• --cpustats is used to provide actual CPU time report (in addition to wall time) in both query log file (for every given query) and status report (aggregated). It depends on clock_gettime() Linux system call or falls back to less precise call on certain systems. You might start searchd thus:

bash $ searchd --config /etc/manticoresearch/manticore.conf --cpustats

• --port portnumber (-p for short) is used to specify the port that Manticore should listen on to accept binary protocol requests, usually for debugging purposes. This will usually default to 9312, but sometimes you need to run it on a different port. Specifying it on the command line will override anything specified in the configuration file. The valid range is 0 to 65535, but ports numbered 1024 and below usually require a privileged account in order to run.

An example of usage:

bash $ searchd --port 9313

• --listen ( address ":" port | port | path ) [ ":" protocol ] (or -l for short) Works as --port, but allows you to specify not only the port, but the full path, IP address and port, or Unix-domain socket path that searchd will listen on. In other words, you can specify either an IP address (or hostname) and port number, just a port number, or a Unix socket path. If you specify a port number but not the address, searchd will listen on all network interfaces. A Unix path is identified by a leading slash. As the last parameter, you can also specify a protocol handler (listener) to be used for connections on this socket. Supported protocol values are 'sphinx' and 'mysql' (MySQL protocol used since 4.1).

• --force-preread forbids the server from serving any incoming connection until prereading of table files completes. By default, at startup, the server accepts connections while table files are lazy-loaded into memory. This extends the behavior and makes it wait until the files are loaded.

• --index (--table) <table> (or -i (-t) <table> for short) forces this instance of searchd to only serve the specified table. Like --port, above, this is usually for debugging purposes; more long-term changes would generally be applied to the configuration file itself.

• --strip-path strips the path names from all the file names referenced from the table (stopwords, wordforms, exceptions, etc). This is useful for picking up tables built on another machine with possibly different path layouts.

• --replay-flags=<OPTIONS> switch can be used to specify a list of extra binary log replay options. The supported options are:

  • accept-desc-timestamp, ignore descending transaction timestamps and replay such transactions anyway (the default behavior is to exit with an error).
  • ignore-open-errors, ignore missing binlog files (the default behavior is to exit with an error).
  • ignore-trx-errors, ignore any transaction errors and skip current binlog file (the default behavior is to exit with an error).
  • ignore-all-errors, ignore any errors described above (the default behavior is to exit with an error).

  Example:

  bash $ searchd --replay-flags=accept-desc-timestamp

• --coredump is used to enable saving a core file or a minidump of the server on crash. Disabled by default to speed up of server restart on crash. This is useful for debugging purposes.

bash $ searchd --config /etc/manticoresearch/manticore.conf --coredump

• --new-cluster bootstraps a replication cluster and makes the server a reference node with cluster restart protection. On Linux you can also run manticore_new_cluster. It will start Manticore in --new-cluster mode via systemd.

• --new-cluster-force bootstraps a replication cluster and makes the server a reference node bypassing cluster restart protection. On Linux you can also run manticore_new_cluster --force. It will start Manticore in --new-cluster-force mode via systemd.

Windows options

There are some options for searchd that are specific to Windows platforms, concerning handling as a service, and are only available in Windows binaries.

Note that in Windows searchd will default to --console mode, unless you install it as a service.

• --install installs searchd as a service into the Microsoft Management Console (Control Panel / Administrative Tools / Services). Any other parameters specified on the command line, where --install is specified will also become part of the command line on future starts of the service. For example, as a part of calling searchd, you will likely also need to specify the configuration file with --config, and you would do that as well as specifying --install. Once called, the usual start/stop facilities will become available via the management console, so any methods you could use for starting, stopping and restarting services would also apply to searchd. Example:

bat C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install --config C:\Manticore\manticore.conf

If you want to have the I/O stats every time you start searchd, you need to specify the option on the same line as the --install command thus:

bat C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install --config C:\Manticore\manticore.conf --iostats

• --delete removes the service from the Microsoft Management Console and other places where services are registered, after previously being installed with --install. Note that this does not uninstall the software or delete the tables. It means the service will not be called from the services system, and will not be started on the machine's next start. If currently running as a service, the current instance will not be terminated (until the next reboot or until --stop). If the service was installed with a custom name (with --servicename), the same name will need to be specified with --servicename when calling to uninstall. Example:

bat C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --delete

• --servicename <name> applies the given name to searchd when installing or deleting the service, as it would appear in the Management Console; this will default to searchd, but if being deployed on servers where multiple administrators may log in to the system, or a system with multiple searchd instances, a more descriptive name may be applicable. Note that unless combined with --install or --delete, this option does not do anything. Example:
  bat C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install --config C:\Manticore\manticore.conf --servicename ManticoreSearch

• --ntservice is an option that is passed by the Microsoft Management Console to searchd to invoke it as a service on Windows platforms. It would not normally be necessary to call this directly; this would normally be called by Windows when the service is started, although if you wanted to call this as a regular service from the command-line (as the complement to --console) you could do so in theory.

• --safetrace forces searchd to only use the system's backtrace() call in crash reports. In certain (rare) scenarios, this might be a "safer" way to get that report. This is a debugging option.

• --nodetach switch (Linux only) tells searchd not to detach into the background. This will also cause log entries to be printed out to the console. Query processing operates as usual. This is a debugging option and might also be useful when you run Manticore in a Docker container to capture its output.

Plugin dir

Manticore utilizes the plugin_dir for storing and using Manticore Buddy plugins. By default, this value is accessible to the "manticore" user in a standard installation. However, if you start the searchd daemon manually with a different user, the daemon might not have access to the plugin_dir. To address this problem, ensure you specify a plugin_dir in the common section that the user running the searchd daemon can write to.

Signals

searchd supports a number of signals:

• SIGTERM - Initiates a clean shutdown. New queries will not be handled, but queries that are already started will not be forcibly interrupted.
• SIGHUP - Initiates tables rotation. Depending on the value of seamless_rotate setting, new queries might be shortly stalled; clients will receive temporary errors.
• SIGUSR1 - Forces reopen of searchd log and query log files, allowing for log file rotation.

Environment variables

• MANTICORE_TRACK_DAEMON_SHUTDOWN=1 enables detailed logging while searchd is shutting down. It's useful in case of some shutdown problems, such as when Manticore takes too long to shut down or freezes during the shutdown process.

The image is based on current release of Manticore package.

The default configuration includes a sample Real-Time table and listens on the default ports:

• 9306 for connections from a MySQL client
• 9308 for connections via HTTP
• 9312 for connections via a binary protocol (e.g. in case you run a cluster)

The image comes with libraries for easy indexing data from MySQL, PostgreSQL, XML and CSV files.

How to run Manticore Search Docker image

Quick usage

The below is the simplest way to start Manticore in a container and log in to it via the mysql client:

docker run -e EXTRA=1 --name manticore --rm -d manticoresearch/manticore && echo "Waiting for Manticore docker to start. Consider mapping the data_dir to make it start faster next time" && until docker logs manticore 2>&1 | grep -q "accepting connections"; do sleep 1; echo -n .; done && echo && docker exec -it manticore mysql && docker stop manticore

Note that upon exiting the MySQL client, the Manticore container will be stopped and removed, resulting in no saved data. For information on using Manticore in a production environment, please see below.

The image comes with a sample table that can be loaded like this:

mysql> source /sandbox.sql

Also, the mysql client has several sample queries in its history that you can run on the above table, just use Up/Down keys in the client to see and run them.

Production use

Ports and mounting points

For data persistence the folder /var/lib/manticore/ should be mounted to local storage or other desired storage engine.

The configuration file inside the instance is located at /etc/manticoresearch/manticore.conf. For custom settings, this file should be mounted to your own configuration file.

The ports are 9306/9308/9312 for SQL/HTTP/Binary, expose them depending on how you are going to use Manticore. For example:

docker run -e EXTRA=1 --name manticore -v $(pwd)/data:/var/lib/manticore -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

or

docker run -e EXTRA=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -v $(pwd)/data:/var/lib/manticore/ -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

Make sure to remove 127.0.0.1: if you want the ports to be available for external hosts.

Manticore Columnar Library and Manticore Buddy

The Manticore Search Docker image doesn't come with the Manticore Columnar Library pre-installed, which is necessary if you require columnar storage and secondary indexes. However, it can easily be enabled during runtime by setting the environment variable EXTRA=1. For example, docker run -e EXTRA=1 ... manticoresearch/manticore. This will download and install the library in the data directory (which is typically mapped as a volume in production environments) and it won't be re-downloaded unless the Manticore Search version is changed.

Using EXTRA=1 also activates Manticore Buddy, which is used for processing certain commands. For more information, refer to the changelog.

If you only need the MCL, you can use the environment variable MCL=1.

Docker-compose

In many cases, you may want to use Manticore in conjunction with other images specified in a Docker Compose YAML file. Below is the minimal recommended configuration for Manticore Search in a docker-compose.yml file:

version: '2.2'

services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
    restart: always
    ports:
      - 127.0.0.1:9306:9306
      - 127.0.0.1:9308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data:/var/lib/manticore
#      - ./manticore.conf:/etc/manticoresearch/manticore.conf # uncomment if you use a custom config

Besides using the exposed ports 9306 and 9308, you can log into the instance by running docker-compose exec manticore mysql.

HTTP protocol

HTTP protocol is exposed on port 9308. You can map the port locally and connect using curl.:

docker run -e EXTRA=1 --name manticore -p 9308:9308 -d manticoresearch/manticore

Create a table:

POST /cli -d 'CREATE TABLE testrt ( title text, content text, gid integer)'

Insert a document:

POST /insert
-d'{"index":"testrt","id":1,"doc":{"title":"Hello","content":"world","gid":1}}'

Perform a simple search:

POST /search -d '{"index":"testrt","query":{"match":{"*":"hello world"}}}'

Logging

By default, the server is set to send its logging to /dev/stdout, which can be viewed from the host with:

docker logs manticore

The query log can be diverted to Docker log by passing the variable QUERY_LOG_TO_STDOUT=true.

Multi-node cluster with replication

Here is a simple docker-compose.yml for defining a two node cluster:

version: '2.2'

services:

  manticore-1:
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
    restart: always
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    networks:
      - manticore
  manticore-2:
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
    restart: always
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
      memlock:
        soft: -1
        hard: -1
    networks:
      - manticore
networks:
  manticore:
    driver: bridge

• Start it: docker-compose up
• Create a cluster:
  ```sql
  $ docker-compose exec manticore-1 mysql

mysql> CREATE TABLE testrt ( title text, content text, gid integer);

mysql> CREATE CLUSTER posts;
Query OK, 0 rows affected (0.24 sec)

mysql> ALTER CLUSTER posts ADD testrt;
Query OK, 0 rows affected (0.07 sec)

MySQL [(none)]> exit
Bye
```

• Join to the the cluster on the 2nd instance
  ```sql
  $ docker-compose exec manticore-2 mysql

mysql> JOIN CLUSTER posts AT 'manticore-1:9312';
mysql> INSERT INTO posts:testrt(title,content,gid) VALUES('hello','world',1);
Query OK, 1 row affected (0.00 sec)

MySQL [(none)]> exit
Bye
```

• If you now go back to the first instance you'll see the new record:
  ```sql
  $ docker-compose exec manticore-1 mysql

MySQL [(none)]> select * from testrt;
+---------------------+------+-------+---------+
| id | gid | title | content |
+---------------------+------+-------+---------+
| 3891565839006040065 | 1 | hello | world |
+---------------------+------+-------+---------+
1 row in set (0.00 sec)

MySQL [(none)]> exit
Bye
```

Memory locking and limits

It's recommended to overwrite the default ulimits of docker for the Manticore instance:

 --ulimit nofile=65536:65536

For best performance, table components can be "mlocked" into memory. When Manticore is run under Docker, the instance requires additional privileges to allow memory locking. The following options must be added when running the instance:

  --cap-add=IPC_LOCK --ulimit memlock=-1:-1

Configuring Manticore Search with Docker

If you want to run Manticore with a custom configuration that includes table definitions, you will need to mount the configuration to the instance:

docker run -e EXTRA=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -v $(pwd)/data/:/var/lib/manticore -p 127.0.0.1:9306:9306 -d manticoresearch/manticore

Take into account that Manticore search inside the container is run under user manticore. Performing operations with table files (like creating or rotating plain tables) should be also done under manticore. Otherwise the files will be created under root and the search daemon won't have rights to open them. For example here is how you can rotate all tables:

docker exec -it manticore gosu manticore indexer --all --rotate

You can also set individual searchd and common configuration settings using Docker environment variables.

The settings must be prefixed with their section name, example for in case of mysql_version_string the variable must be named searchd_mysql_version_string:

docker run -e EXTRA=1 --name manticore  -p 127.0.0.1:9306:9306  -e searchd_mysql_version_string='5.5.0' -d manticoresearch/manticore

In case of the listen directive, new listening interfaces using the Docker variable searchd_listen in addition to the default ones. Multiple interfaces can be declared, separated by a semi-colon ("|"). To listen only on a network address, the $ip (retrieved internally from hostname -i) can be used as address alias.

For example -e searchd_listen='9316:http|9307:mysql|$ip:5443:mysql_vip' will add an additional SQL interface on port 9307, an SQL VIP listener on port 5443 running only on the instance's IP, and an HTTP listener on port 9316, in addition to the defaults on 9306 and 9308, respectively.

$ docker run -e EXTRA=1 --rm -p 1188:9307  -e searchd_mysql_version_string='5.5.0' -e searchd_listen='9316:http|9307:mysql|$ip:5443:mysql_vip'  manticore
[Mon Aug 17 07:31:58.719 2020] [1] using config file '/etc/manticoresearch/manticore.conf' (9130 chars)...
listening on all interfaces for http, port=9316
listening on all interfaces for mysql, port=9307
listening on 172.17.0.17:5443 for VIP mysql
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on 172.17.0.17:9312 for sphinx
listening on all interfaces for http, port=9308
prereading 0 indexes
prereaded 0 indexes in 0.000 sec
accepting connections

Startup flags

To start Manticore with custom startup flags, specify them as arguments when using docker run. Ensure you do not include the searchd command and include the --nodetach flag. Here's an example:

docker run -e EXTRA=1 --name manticore --rm manticoresearch/manticore:latest --replay-flags=ignore-trx-errors --nodetach

Running under non-root

By default, the main Manticore process searchd is running under user manticore inside the container, but the script which runs on starting the container is run under your default docker user which in most cases is root. If that's not what you want you can use docker ... --user manticore or user: manticore in docker compose yaml to make everything run under manticore. Read below about possible volume permissions issue you can get and how to solve it.

Creating plain tables on startup

To build plain tables specified in your custom configuration file, you can use the CREATE_PLAIN_TABLES=1 environment variable. It will execute indexer --all before Manticore starts. This is useful if you don't use volumes, and your tables are easy to recreate.

docker run -e CREATE_PLAIN_TABLES=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -p 9306:9306 -p 9308:9308 -d manticoresearch/manticore

Troubleshooting

Permissions issue with a mounted volume

In case you are running Manticore Search docker under non-root (using docker ... --user manticore or user: manticore in docker compose yaml), you can face a permissions issue, for example:

FATAL: directory /var/lib/manticore write error: failed to open /var/lib/manticore/tmp: Permission denied

or in case you are using -e EXTRA=1:

mkdir: cannot create directory ‘/var/lib/manticore/.mcl/’: Permission denied

This can happen because the user which is used to run processes inside the container may have no permissions to modify the directory you have mounted to the container. To fix it you can chown or chmod the mounted directory. If you run the container under user manticore you need to do:

chown -R 999:999 data

since user manticore has ID 999 inside the container.

On Windows, if you want Manticore to start at boot, you can install it as a Windows Service. You can follow the instructions in the Manticore as Windows Service guide to install Manticore as a service.

Once Manticore is installed as a service, you can start and stop it from the Control Panel or from the command line using the sc.exe command.

sc.exe start Manticore

sc.exe stop Manticore

Alternatively, if you don't install Manticore as a Windows service, you can start it from the command line by running the following command:

.\bin\searchd -c manticore.conf

This command assumes that you have the Manticore's binary and the configuration file in the current directory.

Starting Manticore via HomeBrew package manager

If Manticore is installed using HomeBrew, you can run it as a Brew service.

To start Manticore, run the following command:

brew services start manticoresearch

To stop Manticore, run the following command:

brew services stop manticoresearch

Full-text fields and attributes

Manticore's data types can be split into two categories: full-text fields and attributes.

Full-text fields

Full-text fields:

• can be indexed with natural language processing algorithms, therefore can be searched for keywords
• cannot be used for sorting or grouping
• original document's content can be retrieved
• original document's content can be used for highlighting

Full-text fields are represented by the data type text. All other data types are called "attributes".

Attributes

Attributes are non-full-text values associated with each document that can be used to perform non-full-text filtering, sorting and grouping during a search.

It is often desired to process full-text search results based not only on matching document ID and its rank, but also on a number of other per-document values. For example, one might need to sort news search results by date and then relevance, or search through products within a specified price range, or limit a blog search to posts made by selected users, or group results by month. To do this efficiently, Manticore enables not only full-text fields, but also additional attributes to be added to each document. These attributes can be used to filter, sort, or group full-text matches, or to search only by attributes.

The attributes, unlike full-text fields, are not full-text indexed. They are stored in the table, but it is not possible to search them as full-text.

A good example for attributes would be a forum posts table. Assume that only the title and content fields need to be full-text searchable - but that sometimes it is also required to limit search to a certain author or a sub-forum (i.e., search only those rows that have some specific values of author_id or forum_id); or to sort matches by post_date column; or to group matching posts by month of the post_date and calculate per-group match counts.

CREATE TABLE forum(title text, content text, author_id int, forum_id int, post_date timestamp);

POST /cli -d "CREATE TABLE forum(title text, content text, author_id int, forum_id int, post_date timestamp)"

$index = new \Manticoresearch\Index($client);
$index->setName('forum');
$index->create([
    'title'=>['type'=>'text'],
    'content'=>['type'=>'text'],
    'author_id'=>['type'=>'int'],
    'forum_id'=>['type'=>'int'],
    'post_date'=>['type'=>'timestamp']
]);

utilsApi.sql('CREATE TABLE forum(title text, content text, author_id int, forum_id int, post_date timestamp)')

res = await utilsApi.sql('CREATE TABLE forum(title text, content text, author_id int, forum_id int, post_date timestamp)');

utilsApi.sql("CREATE TABLE forum(title text, content text, author_id int, forum_id int, post_date timestamp)");

utilsApi.Sql("CREATE TABLE forum(title text, content text, author_id int, forum_id int, post_date timestamp)");

table forum
{
    type = rt
    path = forum

    # when configuring fields via config, they are indexed (and not stored) by default
    rt_field = title
    rt_field = content

    # this option needs to be specified for the field to be stored
    stored_fields = title, content

    rt_attr_uint = author_id
    rt_attr_uint = forum_id
    rt_attr_timestamp = post_date
}

This example shows running a full-text query filtered by author_id, forum_id and sorted by post_date.

select * from forum where author_id=123 and forum_id in (1,3,7) order by post_date desc

POST /search
{
  "index": "forum",
  "query":
  {
    "match_all": {},
    "bool":
    {
      "must":
      [
        { "equals": { "author_id": 123 } },
        { "in": { "forum_id": [1,3,7] } }
      ]
    }
  },
  "sort": [ { "post_date": "desc" } ]
}

$client->search([
        'index' => 'forum',
        'query' =>
        [
            'match_all' => [],
            'bool' => [
                'must' => [
                    'equals' => ['author_id' => 123],
                    'in' => [
                        'forum_id' => [
                            1,3,7
                        ]
                    ]
                ]
            ]
        ],
        'sort' => [
        ['post_date' => 'desc']
    ]
]);

searchApi.search({"index":"forum","query":{"match_all":{},"bool":{"must":[{"equals":{"author_id":123}},{"in":{"forum_id":[1,3,7]}}]}},"sort":[{"post_date":"desc"}]})

res = await searchApi.search({"index":"forum","query":{"match_all":{},"bool":{"must":[{"equals":{"author_id":123}},{"in":{"forum_id":[1,3,7]}}]}},"sort":[{"post_date":"desc"}]});

HashMap<String,Object> filters = new HashMap<String,Object>(){{
    put("must", new HashMap<String,Object>(){{
        put("equals",new HashMap<String,Integer>(){{
            put("author_id",123);
        }});
        put("in",
            new HashMap<String,Object>(){{
                put("forum_id",new int[] {1,3,7});
        }});
    }});
}};
Map<String,Object> query = new HashMap<String,Object>();
query.put("match_all",null);
query.put("bool",filters);
SearchRequest searchRequest = new SearchRequest();
searchRequest.setIndex("forum");
searchRequest.setQuery(query);
searchRequest.setSort(new ArrayList<Object>(){{
    add(new HashMap<String,String>(){{ put("post_date","desc");}});
}});
SearchResponse searchResponse = searchApi.search(searchRequest);